I don't know. . . In the context, it seems like the "men" he is referring to are "brothers."
The story starts out with how those with an earthly hope - JWs, not "worldly folks" - have the "very important" role of supporting Christ's brothers.
It goes without saying that the rest of the world have no role in the first place. They could care less.
Then he goes on to the story about being dissed in a "branch territory." Again he could have said he was poorly treated in "another country" but specifically used the term "branch."
It really wouldn't make sense if it were just "worldlies" who are pissing him off - that's to be expected; and they're all burnt hot dogs via Big Brother Jesus' firebolts anyway.
However, it does seem he is making the point that those with an earthly hope - the ones who are supposed to support him- are the ones who dissed him, and thus they will have to deal with JC.
Just my take on it.